Methods and systems for embedding camera information in images

ABSTRACT

The present invention is related to providing, in association with an image, information related to the image capture device used to capture the image. In one embodiment, information related to a first static camera characteristic and camera setting information related to a first captured digitized image is embedded in the first captured digitized image using a watermark. The watermarked information may be used to identify the source or owner of the picture, and/or to aid in the more accurate reproduction of the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation application of U.S. patent application Ser. No.12/657,704, filed Jan. 26, 2010, which is a divisional application ofU.S. patent application Ser. No. 10/067,463, filed Feb. 4, 2002, nowU.S. Pat. No. 7,663,670, issued Feb. 16, 2010, which claims benefit ofU.S. Provisional Patent Application No. 60/268,134, filed Feb. 9, 2001,all of which are herein incorporated by reference in their entirety.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention is related to digital images, and in particular,to embedding camera-related information in digital images.

(2) Description of the Prior Art

Digital media, such as digital photographs and video, has becomeincreasingly prevalent as the performance of digital cameras and colorprinters have improved and their cost decreased. In addition, thepopularity of the Internet, which facilitates the distribution, viae-mail or Web sites, of digital images has also increased the creationof 15 digital media.

However, images taken using conventional digital media capture devices,such as conventional digital cameras, typically are not adequatelyreproduced, whether the reproduction is accomplished using an ink jetprinter, a color laser jet printer, a thermal wax printer or on acomputer display. This failure to adequately reproduce the capturedimages may be attributed in part to the fact that image capture deviceperformance characteristics are not taken into account. In addition,little or no information is provided by the image capture device on thecamera settings used to capture a given image. Further, even when verylimited information is provided, the information is provided as aseparate file, which is associated with the given image. Thus, theinformation is not provided as part of a standard image file, such as aJPEG image file.

Because of the deficiencies noted above, users of digital image capturedevices are still often dissatisfied with their performance and theresulting images reproductions.

SUMMARY OF THE INVENTION

Various embodiments of the present invention are directed to providingfor the enhancement of image reproductions and the authentication of thesources of images.

Thus, one embodiment of the present invention embeds image capturecharacteristics and/or settings into images capture by the image capturedevice.

For example, if the image capture device is a digital camera, thedigital camera performance or characteristics information may includecolor correction information, white balance information, filterinformation, bad pixel information, and/or correction information andcorrection algorithms for the foregoing. Other types of digital cameraperformance or characteristics information may include leakage currentsassociated with the camera imaging sensor, the sensor's dynamic range,the sensor's sensitivity, and so on.

Further, one embodiment of the present invention embeds camera settinginformation associated with capturing a given image in the given image.For example, the camera setting information may include shutter speed,aperture settings, flash information, such as whether a flash was usedand the strength of the flash, the ambient light conditions, and so on.By providing camera setting information, as well as performance orcharacteristics information, the reproductions of images can be madewith greater accuracy. In addition, the image reproductions can beoptimized based on the type of image reproductions technology beingused. For example, setting and camera characteristic information can beused to optimize the image reproduction process selectively for inkjetprinter reproductions, color laser printer reproductions, andreproductions intended to be viewed on a monitor.

One embodiment of the present invention can embed in images authorshipand/or source information. For example, a user identifier associatedwith the image taker may be embedded in corresponding images. Inaddition, information identifying the image capture device, such as aserial number or substantially unique camera characteristics, may beembedded in images captured with the image capture device. This allowsimage ownership to be better tracked, allowing for more securedistribution of images and better ensuring that copyright holders arecompensated for their work. Further, the use of identifiers or otherownership and/or source information enables the use, transfer and/orsale of captured images to be metered and further enables an automatedaccounting process for tracking the sale or use of images and assessingcharges accordingly. Thus, image owners can be appropriately compensatedfor the use of their images, and image owners are incentivized to postimages for use by others.

In one embodiment of the present invention, some or all of theinformation described above is embedded into images using a digitalwatermark, though other techniques, such as the use of a separate filecontaining the information, may be used as well. The separate file maycontain sets of other data as well. The digital watermark may beembedded using, by way of example, either spatial domain techniques orfrequency domain techniques. The watermark characteristics may be chosensuch so that the watermark will be unobtrusive, that is, not easilyperceptible to the unaided eye, robust, that is difficult to remove,and/or unambiguously identify the source or owner of the image. Thewatermark may be inserted before compression in the base band, or afterimage compression. If the watermark is inserted before imagecompression, then the watermark implementation is such as to besubstantially preserved in the compressed image.

The watermark may optionally be created using a novel key seedtechnique, wherein the seed is related to or based at least in part onone or more selected characteristics of the corresponding image capturedevice. Image capture device characteristics may be selected on theirlikely uniqueness. These characteristics may include, by way of example,pixel gain values, pixel offset values, device leakage currents, thenumber and/or locations of defective pixels, and so on.

One embodiment of the present invention is a method of embedding camerainformation and image capture related information in a digital form ofan image, comprising receiving information on a first static cameracharacteristic, receiving camera setting information related to a firstcaptured digitized image, generating an encryption key based at least inpart on the first static camera characteristic, embedding a watermark inthe first captured digitized image, wherein the watermark contains atleast a portion of the information on the first static characteristicand at least a portion of the camera setting information related to thefirst captured digitized image, and encrypting the watermark using theencryption key. The first static camera characteristic may be, by way ofexample, a camera image sensor bad pixel characteristic, may be relatedtc^(˜) sensor current value, or may be related to a camera image sensorsensitivity. The camera setting information can include informationrelated to the flash intensity used to capture the first captureddigitized image. The method optionally includes information in thewatermark related to the ambient light present when the image wascaptured by the camera and/or includes at least a first dynamicallymeasured camera characteristic.

Another aspect of the present invention is a digital camera systemcomprising an imager, a first static camera characteristic associatedwith the imager, a first variable camera setting, a watermark generatorused to embed in the form of a watermark at least one of the firststatic camera characteristic and the first variable camera settinginformation in an image captured by the camera, and a key generatorconfigured to generate an encryption key used to encrypt the watermark.The watermark generator may be configured to generate watermarks thatare visually perceptible or create watermarks that are visuallyimperceptible. The first variable camera setting may be, by way ofexample, a shutter speed, an aperture setting, or a flash setting. Thefirst static camera characteristic may be, by way of example, related toan imager current, to defective pixels associated with the imager, orimager gamma information.

Still another aspect of the present invention is a method of extractingcamera-related information from an image captured by the camera, themethod comprising receiving an image, locating a watermark in the image,extracting from the watermark camera characteristic information andcamera setting information associated with the camera used to capturethe image, and enhancing the image based at least in part on theextracted camera characteristic information and camera settinginformation. Optionally, the method further comprises enhancing theimage specifically for a reproduction device which will be used togenerate a reproduction of the enhanced image. By way of example, thecamera characteristic information includes a pointer into a gammacorrection lookup table. The lookup table can be stored external to thecamera. The camera characteristic information can optionally include Fnumber-related information.

The method optionally further comprises selecting a filter algorithmbased at least in part on the F-number related information. Anotheraspect of the present invention is decrypting the watermark utilizinginformation associated with the camera.

The information associated with the camera is, in one embodiment,related to at least the number of bad pixels of an imager included inthe camera.

Still another aspect of the present invention is a method of extractingcamera-related information from an image captured by the camera, themethod comprising receiving an image, locating a watermark in the image,extracting camera setting information associated with the camera used tocapture the image from the watermark, receiving information relating tothe gamma of a targeted reproduction device, and enhancing reproductionsof the image made by the targeted reproduction device based at least inpart on the extracted camera setting information and the gamma relatedinformation.

One aspect of the present invention is a method of including camerainformation and image capture related information in association with adigital form of an image, the method comprising, capturing an image,digitizing the image, receiving information on a first static cameracharacteristic, inserting in a data set associated with the digitizedimage at least a portion of the information on the first staticcharacteristic, and transmitting the digitized image and the data set toan image processor.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings forming a material part of thisdescription, there is shown:

FIGS. 1A-B illustrate an example of a digital image capture device whichmay be used with one embodiment of the present invention;

FIG. 2 illustrates an example system for embedding a watermark in adigital image;

FIG. 3 illustrates an example process for embedding a watermark in thefrequency domain of a digital image;

FIG. 4 illustrates an example method of embedding camera and imagecapture related information in an image information and retrieving theembedded information;

FIGS. 5A-F illustrate example information which may be embedded into ortransferred with an image; and

FIGS. 6A-B illustrate example file headers used to transfer camera andimage capture related information with the image.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

One embodiment of the present invention is directed to including orembedding imaging device characteristics information and/or camerasetting information in or in association with images taken with theimaging device. This information may later be used to reproduce theimages with greater fidelity and accuracy than would be otherwisepractically achieved.

Further, as will be described in greater detail below, the aboveinformation, as well as source and ownership information, may beembedded in images using a digital watermark or signature.Advantageously, a strong key may be created to embed the watermark orsignature using the imaging device's characteristics as seed data usedto produce the key.

One embodiment of the present invention will now be described in greaterdetail with reference to FIGS. 1-6. Throughout the followingdescription, reference will be made to various implementation-specificdetails, including, for example, image device types and architectures,encoding and compression standards, reproduction devices, watermarkingtechniques, and image enhancement techniques. These details are providedin order to fully set forth a preferred embodiment of the invention, andnot to limit the scope of the invention. The scope of the invention isset forth in the appended claims.

FIG. 1 illustrates an example imaging device in the form of a digitalcamera 100 which may be used with one embodiment of the presentinvention. Of course other imaging devices, such a video camera, may beused as well. In the illustrated example, the digital camera 100includes an imager frontend 102, a backend interface 120, a backendprocessing section 122, and a host interface 156.

The imager frontend 102 includes an imager in the form of an imagesensor array 104. The image sensor array 104 may be, by way of example,a CMOS or a CCD array. If the image sensor array 104 is implementedusing CMOS technology, then other digital circuitry, such as some or allof the frontend 102 or backend processing 122 circuitry, can be includedon the same substrate as the image sensor array 104. The image sensorarray 104 contains an array of pixels which converts photons intoelectrical signals. Typical digital camera image sensors currently havebetween 1 and 3 megapixels, though they may have fewer or more pixels.Overlying the sensor array 104 is a color filter array and the lenssystem used to focus light onto the sensor. The lens system may have afixed or variable focus length. The lens system may optional have zoomcapabilities. In addition, the example imager front end 102 includes afixed pattern noise subtractor 106, an analog compression system 108, ananalog to digital converter 110, and a bad pixel correction circuit 112.

The image sensor output is provided in the form of one or more analogsignals to the fixed pattern noise subtractor 106 which subtracts orremoves known noise inherent in the image sensor. Variation betweenpixels-to pixel voltage fluctuation is a principle source for fixedpattern noise. Standard Correlated Double Sampling (CDS) techniques aredeployed to reduce to this type of noise.

Another source of noise results from dark currents, that is, leakagecurrent which is present even if no light is present. This type of noisemay be compensated through the use of offsets which compensate for thedark currents. The resulting corrected signal undergoes analogcompression by the analog compression circuit 108. The compressed signalis then converted into a digital signal by the A/D converter 110. Thedigital signal is provided to a bad pixel correction circuit 112 whichdetects and/or corrects for bad or defective pixels, that is, pixelswhose response is outside of a desired range. Bad pixels may be due toprocess non-uniformity and yield issues.

Once the bad pixel correction is performed, the resulting digitalinformation is transferred via an interface 118, which may be a parallelor serial interface, to the backend processing system 122. In addition,information from other sensors, such as ambient light sensors andfocusing sensors, may also to provided to the backend system. Inresponse to sensor readings, the backend processing and controlcircuitry appropriately controls the camera aperture settings, shutterspeed, flash, and lens focusing.

The digitized, corrected image received over the parallel interface 118is transformed into the linear domain by an inverse compressor 124,which provides mapping enhancement for the inverted pixel non-linearity.In response to the focus sensor and light sensor readings, the focus 126and exposure 128 systems automatically moves the camera lens to create asharp focused image on the imager array 104. Typically, the exposuretime is controlled and manipulated by adjusting the integration time andthe gain level.

Color crosstalk, which is caused by electrons diffusing into neighboringpixels so that a pixel of a specific color suffers a reduction in colorsensitivity and “pollutes” neighboring pixels of different colors,results in overall reduced color fidelity. The color crosstalk iscorrected by a color reduction crosstalk system 130.

A color interpolation system 132 reconstructs the missing colorcomponents due to color sub-sampling by the color filter layer. A colorcorrection system 134 compensates for deficiencies in the color responseof the sensor 104. A gamma correction system 136 inverts the imagesensor array exponential response. A color space conversion system 138converts image color information into a desired color space, such asdevice dependents color spaces or device independent color spaces.Examples of device dependent color spaces are CMYK and RGB. CMYKspecifies the amounts of ink needed to reproduce an ink-based copy ofthe image. In the RGB color space, the RGB values are expressed asdigitized video voltages. Device independent color spaces include CIELABor CIEXYZ. A white balance circuit 140 adjusts the color gain accordingthe ambient light.

Enhancement filters 142 are used to sharpen or smooth the image. Forexample, a typical enhance function attenuates the low frequency contentof the image and/or emphasizes the high frequency content, therebysharpening the image. The sharpening function may be performed using avariety of methods, including basic linear filtering using a 3×3 FIRfilter or using a more complex non-linear filtering such as conditionalun-sharp masking.

A compression system 144 compresses the image data so to fit in asmaller 25 memory size and/or to meet the bandwidth requirements of thehost or other interface.

Typically, in order to achieve a smaller image file size, a lossycompression is employed. However, typically, digital cameras allow auser to choose the tradeoff level between image quality and image filesize by enabling compression and selecting the degree of compression orby disabling lossy compression altogether. A commonly used lossycompression for a still image is JPEG. The JPEG compression processcompresses the image in the frequency domain. Frequency domainrepresentation of an image are generally more compact than the spatialdomain representation since the frequency transform exploits periodicityand thus reduces or eliminates redundancies. Insignificant imagecomponents appear in frequencies having low amplitude. The lossy steptruncates these components down to a selected level. The quality of therestored image is generally directly proportional to the truncationlevel. Using discrete cosine transform, JPEG provides lossy compressionwith ratios up to 100:1 and higher. Images are typically compressed inratios of 10:1 to 20:1 to minimize noticeable loss. The more the losscan be tolerated, the more the image can be compressed. Compression isachieved by dividing the picture into tiny pixel blocks, which arehalved over and over until the ratio is achieved. Other standard fileformats which may be used include, by way of example, MPEG, TIFF, GIF,and JPEG-lite formats. In addition, a custom or vendor specific fileformat may be used as well.

After the compression takes place, a watermarking system 146 inserts avisually perceptible or imperceptible signal or data, termed a“watermark,” into an image. In one embodiment, the watermarkcharacteristics are selected so that the watermark will be unobtrusive,that is, not easily perceptible to the unaided eye, robust, that isdifficult to remove, and/or unambiguously identify the source or ownerof the image. The data contained within the watermark can later beretrieved and/or authenticated. The watermark encoding of the image canoccur either in the base band domain or the compressed domain. Variouswatermarking techniques can be used within the scope of the presentinvention. For example, the watermark can also be embedded before theimage compression is performed. However, it is desirable that thewatermark be robust enough to adequately survive at least commoncompression processes.

The watermarked image may then be transferred to a^(˜) targeted displayor print 25 device via an appropriate host interface 156 port, such asan NTSC/PAL port 158 for display on a television, an JrDa infrared port160 for transfer to an appropriately equipped computer, PDA, cellularphone, or printer, an RS-232 serial port 162, a parallel port 164, aBluetooth port (not shown), a modem port (not shown), or a USB port 168.All or portions of the backend system 122 may be located within thecamera 100, in a separate portable unit, in a personal computer, or adedicated processing system. In addition some of the backend functionsmay be duplicated so that they may be performed both within and externalto the camera. Thus, for example a user can select to have the rawfrontend digitized image provided to an external system for processing.Because external systems are not constrained by the same portabilityconsiderations as handheld cameras, the external systems can havegreater processing power and can perform relatively better imageprocessing. Alternatively, the user may elect to have the imageprocessed at least partially within the camera and have the processedimage output from the camera. Optionally, the user may elect to haveboth the raw digitized image and the processed image output.

In addition to the processing systems described above, the digitalcamera 100 typically includes timing generation circuitry 114 that isresponsible for producing the necessary clocks, synchronization signals,a circuit for reading the pixel data, and registers for controlling theimage sensor array 104 operation. The registers may be set via aninterface 116, which may be a serial or parallel interface. Non-volatilememory 148 is provided to store camera characteristics information, asdescribed below, as well as to store embedded code used to operate thecamera. A user can control the operation of the camera 100 via a userinterface 152 and LCD control 150.

The systems described above can be implemented using one or moreintegrated circuits and/or software or firmware executing on one or moreintegrated circuits. The integrated circuits may include signalprocessing circuits or processors, general purpose microprocessors ormicrocontrollers, custom processors, and/or state machines.

With respect to the present invention, the particular techniques orsystems used to perform the various corrections and enhancements are notcritical. For example, many of the correction functions can be performedon the fly, using circuitry tightly coupled to and optionally located onthe same substrate as the sensor array 104. Examples of such circuitryand techniques are discussed in co-pending applications, U.S.Provisional Application No. 60/149,796 filed Aug. 19, 1999, U.S.application Ser. Nos. 09/495,971, 09/496,364, 09/496,533, 09/496,607,filed Feb. 2, 2000, 2000, and U.S. Provisional application Ser. No.09/659,355, filed Sep. 12, 2000, which are incorporated herein byreference in their entirety.

Information about the digital camera's frontend characteristics areprovided within watermarks embedded into photographs, included in theimage file header, stored in a separate file, and/or otherwise providedwith photographs taken with the digital camera. For example, sensorinformation, such as the type of sensor, the frequency response of theimage sensor, color crosstalk information, pixel gains, pixel offsets,pixel column offsets, pixel row offsets, and/or color information. Thecolor information can include color gains for red, green, and bluepixels if a primary color filer is used, and color gains for cyan,magenta, and yellow pixels if a complementary color filer is used. Noiseinformation, such as dark current and leakage current information, mayalso be included in the watermark. The frontend characteristics, as wellas other camera characteristics, may be measured during themanufacturing phase and stored in the nonvolatile memory 148, which maybe in the form of ROM, EPROM, UVPROM, disk drive, or other memory type.This information will then be static, that is, the stored informationwill not generally change for each image. The information may then bereadout for inclusion in the watermark. In another embodiment, as willbe understood by one of ordinary skill art, some or all of thecharacteristics may be measured dynamically, on the fly, using standardinstrumentation techniques. For example, currents may be measured usingcurrent shunts, and the measurements may then be converted into adigital format using an analog-to-digital converter.

In addition, camera setting and lighting information, corresponding to agiven photograph, may also be included in the watermark for thatphotograph. For example, the camera setting information may includeshutter speed, aperture settings, flash information, such as whether aflash was used and the strength of the flash. The ambient lightinformation, for example, may indicate the ambient light intensity, thefrequency spectrum of the ambient light, and an indication as to whetherthe ambient light is fluorescent, tungsten or sunlight. The camerasetting information may be provided by the camera controller whichcontrols the settings and by appropriate sensors, such as the ambientlight sensor. The setting information, as well as other informationdiscussed below, may be used to optimize later reproductions of thephotographs, including reproductions made on or using specific CRTs, LCDdisplays, inkjet printers, color laser printers, thermal printers andthe like.

Other useful information may also be included in the watermark orotherwise provided with images taken by the image capture device. Forexample, the image capture device serial number, user information, suchas a user ID, name, and/or contact information, and time and dateinformation, may be included in the watermark. Color information, suchas color gains for Red, Blue and Green pixels in the case of primarycolor filters, and Cyan, Magenta and Yellow for complementary colorfilters, may be included as well. The color information may includecolor correction matrices. FIG. 5A illustrates a correction matrix to beused with a primary color filter with RGB data out. FIG. 5B illustratesa correction matrix to be used with a primary color filter with YIN dataout. FIG. 5C illustrates a correction matrix to be used with acomplimentary color filter with RGB data out. The information may alsoinclude white balance information, flash status, the gain and offset ofeach pixel, and so on. By way of example, FIGS. 5D-F illustrate triplet3×3 base matrices used to provide white balance correction for a primarycolor filter with an RGB data out format. The white balance matrix candefine various pixel color types (Bayer pattern or complimentary) aswell as different data output types (RGB, YUV, YCrCb). After selecting adesired data output format, the base matrix can provide fixed valuessetting for different light conditions such as outdoors (FIG. 5D), dusk(FIG. 5E), tungsten (FIG. 5F), and so on. The appropriate colorcorrection and color space conversion matrices illustrated in FIGS. 5A-Fmay be embedded in the watermark or otherwise associated with thecorresponding captured image.

In addition, some or all of the frontend characteristics which, in theaggregate, are substantially unique from camera to camera, may be usedas a unique identifier or as a type of serial number related, at leastin part, to the camera's physical properties. For example, depending onthe size of the image sensor array 104, the array 104 may have between20 to 3000 bad or defective pixels. Because the number of bad pixels andthe distribution pattern of the bad pixels are independently random, thebad pixel information may be used to generate a substantially uniqueidentifier. In addition, as will be described in greater detail below,some or all of the frontend characteristics may also be used to generatea watermark encryption key.

The watermarking process will now be described in greater detail. In aneffort to protect the intellectual property associated with such digitalmedia, digital watermarks haven been used to identify the owner of thedigital media. Digital watermarks are typically used to hide theownership data by embedding watermarks containing the ownership data inthe images using either spatial domain techniques or frequency domaintechniques. Thus, many watermarking techniques suitable for images havebeen developed and, for the purposes of the present invention, theselection of a particular technique is not in any way critical.

FIG. 2 illustrates an example system for embedding a watermark in adigital 10 image. The digital camera 100 captures an image 202. Avariety of camera characteristics and settings 204 that affect thequality and color of the image 202 are embedded in the image in the formof a watermark by the watermark engine 206. In addition, some or all ofthe camera characteristics and settings 204 may be used as a seed by akey generator 204 which generates a key used to encrypt the watermark.

FIG. 3 illustrates one example process flow 300 for embedding awatermark in an image. Some or all of the information discussed abovemay be provided as a payload to be included in the watermark. Thewatermark itself may be encrypted using a key generated using some orall of the frontend characteristics discussed above. For example, apseudo-random number generator 304 may generate a pseudo-random numbersequence w using the key, where the key is based on bad pixelinformation and/or dark current(s) information, as a seed. More than onecamera characteristic may be used to generate the key. For example, thekey may be generated by dividing the image sensor dark current by thenumber of bad pixels. Other camera-related currents or voltages, such asthose discussed above, may be used as well. Thus, a critical problem ingenerating a key meeting two important criteria, that of finding atraceable, unique key, is overcome by using frontend characteristics,which meets both criteria.

A quality operator 306 is used to ensure that when desirable, thewatermark is not perceptible. For example, for a busy picture, thewatermark can be less random, while for a more monochromatic mark thewatermark may be made more random. The watermark is then embedded intothe image. The watermark may be embedded either in the spatial domain orthe frequency domain. By way of example, as is well know to one ofordinary skill in the art, one spatial domain technique embeds thewatermark by changing the value of a pixel LSB (least significant bit)at selected locations. However, spatial domain watermarks may not be asrobust to attacks as frequency domain watermarks. In addition, frequencydomain watermarks may be more likely to survive rotation, scaling andtranslations of images.

If, as illustrated in FIG. 3, the watermark is embedded using spreadspectrum techniques, the image i is transformed into the frequencydomain I at state 302. Coefficient selection is performed at state 310.At state 308, the watermark payload is embedded into the imagecoefficients, which are then summed with the transformed image at state314. The result is inverse transformed at state 312 to create awatermarked version V of the original image, with the payloadinformation contained therein.

FIG. 4 illustrates one embodiment of an end-to-end process 400 ofwatermarking insertion and extraction. The watermark insertion processmay be implemented as described above with reference to FIG. 3, or mayuse other well know techniques. An image x is transformed by awatermarking function 402, resulting in a watermarked version of theimage y. The function 402 also has as inputs a message or payload V andoptionally a key K. If it is desired to keep the watermark payloadcontents secret from the general public, the key K may be a secret key,provided only to authorized persons or commercial entities. As will beunderstood by one of ordinary skill in the art, the watermarked image ymay be distorted by channel “noise” h 404 which may be caused byunintentional distortions or by intentional attacks intending todestroy, degrade, or modify the watermark. The channel output in thisexample is the resulting image z, which is related to the watermarkedimage y by the channel function h 404.

A watermark detector d 408 detects the presence of a watermark in theimage z generated using the key K. If the detector 408 determines thatthe watermark is present, even if in a somewhat distorted or degradedform, using the key K, an extractor e 406 extracts the watermark messageor payload V, or an estimate V′ if the payload is altered.

Depending on the extraction techniques used, the original,un-watermarked image x may or may not be used in the extraction process.For example, a well known technique for detecting and extracting awatermark utilizing the un-watermarked image is described in “SecureSpread Spectrum Watermarking for Multimedia,” I. J. Cox, J. Kilian, T.Leighton and T. Shamoon, Technical Report 95-10, NEC Research Institute.A well know technique for detecting and extracting a watermark withoutusing the original, un-watermarked image is described in “Modulation andinformation hiding in images,” Smith and B. Comiskey, Proceedings ofFirst International Workshop on Information Hiding, Lecture Notes onComputer Science, Cambridge, UK, pp. 207-226, June 1996. Of course otherwatermarking and watermarking extraction techniques may be used as well.

As previously discussed, rather than embedding the information in theimage using a watermark, the camera related data may be included in afile header. FIG. 6A illustrates a standard JPEG hear format. The cameradata may be inserted into the JPEG frame data. FIG. 68 illustrates aproprietary file format, wherein data can be segmented and encrypted ina manner that provides a greater freedom in the management oftransportation as compatibility with the standard file format is notrequired and greater freedom in hiding the data can therefore beprovided. For example the camera data may be included in an Auxiliarydata field.

Once the camera setting and/or information is extracted or retrieved, itcan be used to improve reproductions of the images based on the staticand dynamic camera characteristic information, as well as on thecharacteristics of the targeted display or print system, as follows.

The color correction is typically provided using a color conversionmatrix as described above, or using the coefficient of a polynomial thatconverts the measured image signal that has been corrupted by colorfilter transmission and imager quantum efficiency into a more accurateimage signal level. The color conversion matrix, or coefficient of apolynomial, takes into account gain settings and other imagercharacteristics. Utilizing the color correction information embeddedinto the image, the reproduction of the image may be enhanced in thepost processing application, by optimizing the values of the polynomialand coefficients for the final target device for better matching (e.g.color correction for printers or display monitor).

White balance correction is used to enhance gray areas of images.Whether the camera is selected for auto white balance, or specificsettings, the white balance algorithms take “real” or measured imagedata and converts or modifies the image data to make the gray areasappear more pleasing to the eye. Information on the type of lens filterused if any) and the lighting conditions for a corresponding photographis useful in performing accurate white balance correction. Thus, oneembodiment of the present invention embeds lens filter and lightingcondition information in an image so that the white balance correctionmay be accurately performed off-camera during post-processing. The whitebalance correction may be implemented using a separate white balanceconversion matrix or may be included as part of the color conversionmatrix discussed above.

Gamma correction generally controls the overall brightness of an imageas well as the ratios of red to green to blue. Images which are notproperly corrected may lack contrast or be too dark. Gamma informationis therefore useful in ensuring that image colors are accuratelyreproduced. Thus, it is important to match the imager output with thegamma of the display, such as when the image is to be displayed on a CRTor LCD display or using a video projector.

The gamma information embedded into an image may be in the form of apointer into lookup table containing gamma correction information, suchas the name or location of a gamma correction algorithm suitable for useto correct images captured with the corresponding camera having a givensensor or imager type. Further, the gamma correction algorithm may alsobe selected based at least in part on the device which will be used toreproduce the image. The gamma correction function is typically notperformed equally for all colors or shades of gray within an image.Rather, the gamma correction function is a curve. The gamma functionadjusts the variances between the two extremes (white and black) toachieve a non-linear balance and to correct for conditions which haveaffected the image.

For example, the camera and sensor types or part numbers may be embeddedin the image. Based on the camera and/or sensor type, one or more gammacorrection algorithms or functions may be identified as being suitablefor use with images taken with camera and/or sensor. Then, if more thanone algorithm is identified, the algorithm most appropriate for aspecified or targeted image reproduction device (CRT display, flat paneldisplay, video projector, color laser printer) is selected. This willallow for post processing and optimizing the captured images for thetargeted reproduction device.

By way of example, computer monitors typically have a gamma of 2.5. Thatis, the intensity to voltage response curve for the monitor isapproximately a 2.5 power function. Thus, without gamma correction, ifthe computer monitor received a command that a certain pixel should haveintensity equal to x, where x is between 0 and 1, the monitor willactually display a pixel which has intensity equal to x²⁵, that is lessthan the commanded intensity. Gamma correction is used to adjust thecommanded intensity before the command reaches the monitor so that thedisplayed intensity will accurately reproduce the desired intensity foraccurate image reproduction.

The lookup table information may be stored off-camera, such as on a PCor server used to performed color processing. Thus, different cameramanufactures and image sensors will be looked up from this table andmatched for the correct algorithms.

Average signal level and signal range information, such as the F numberand/or integration time, stored in an image may also be used for postprocessing and optimizing the captured images. By knowing the F numberand the integration time, the average luminance incident can becalculated. In addition, the F number affects the imager opticalcrosstalk which causes blurring or color bleeding in images. Thisinformation can then be used to select the appropriate filter algorithmand to automatically optimize the printing or display of the image. Inaddition, from the stored information, the minimum and maximum luminancemay be calculated and utilized to calibrate displays or printers foroptimum performance

With respect to sharpening algorithms and other filters performed by thebackend processing system, by embedding the information in an image,post processing can optionally remove some or all of the sharpening andfiltering effects to thereby “retrieve” the substantially real ororiginal image data that was present before the sharpening andfiltering. This permits post-processing of the derived raw image datausing more powerful and/or user customized sharpening and filteringtechniques.

With respect to compression techniques or methods, by transmittinginformation about the compression method along with the image,post-processing, performed off-camera can filter or clean up thetransmitted, compressed image. In particular, lossy compressiontechniques, which discard or approximate some of the image information,are amenable to such post-processing.

In addition, as previously discussed, when embedded in an image thestatic camera characteristic information may be used to uniquelyidentify the camera that is the source of the image. The possessor ofthe camera can then demonstrate that the camera was the source of theimage. This may be accomplished by examining the static cameracharacteristics of the camera, such as bad pixel information, colorsensitivities, and so on, and to determine if they match thecorresponding static camera characteristic information embedded in theimage. If they match, then there it is likely that the camera was usedto capture the image.

In another embodiment, rather than examining the camera directly, someor all 15 of the static camera characteristics are stored in a registry,such as the camera manufacturer's or distributor's database. Becausemany of the static camera characteristics are measured by themanufacturer, the manufacturer may efficiently store the static cameracharacteristic information for a given camera in association withanother unique camera identifier, such as a serial number. In addition,if the owner of the camera is known, such as by registration of thecamera by the owner, the owner's identity may also be stored inassociation with the static camera characteristic information. Thus, thetrue camera owner can demonstrate that his or her camera captured theimage by comparing the static camera characteristic informationretrieved from the image with the static camera characteristicinformation stored in the registry. If they match, then there it islikely that the camera was used to capture the image.

In one embodiment, rather than embedding static camera characteristicinformation in an image, the camera serial number or other cameraidentifier may be embedded in the image. Once the identifier isretrieved, the some or all of the static camera characteristicinformation is received from the camera vendor or manufacturer.

The static camera characteristic information may be received from a Website, optical disk, or other site or storage device. The received staticcamera characteristic information may then be used to enhance oroptimize images and image reproductions as previously discussed.

As previously described, camera identifiers, authorship and/or sourceinformation can be imbedded in images using watermarks. For example, auser identifier associated with the image taker may be embedded incorresponding images “authored” by the image taker. Further, a useridentifier associated with an image owner, who may also be thephotographer who captured the image, may be embedded in correspondingimages. The image owner may be the owner of the image copyright, or alicensee of the copyright with a right to sublicense. In addition,information identifying the image capture device, such as a serialnumber or substantially unique camera characteristics, may be embeddedin images captured with the image capture device. This allows imageownership to be better tracked, allowing for more secure distribution ofimages and better ensuring that copyright holders are compensated fortheir work.

Further, embedding camera identifiers, ownership and/or authorshipinformation 15 images provides the ability to meter the use or transferof the captured images. Further, using the embedded or watermarkedidentifiers, an automated accounting process tracks the sale or use ofimages and assesses, credits, and debits charges accordingly. Asdiscussed in greater detail below, to further enable the automation ofthe accounting and payment process, each image may have embedded in it aunique image identifier.

For example, if images are posted to a Web site or exchanged via a Website, an accounting application executing on or in conjunction with theWeb site tracks image downloads from the Web site and exchanges of theimages via the Web site by users utilizing Internet terminals or thelike. The accounting application, using the extraction system describedabove with respect to FIG. 4 or the like, extracts from the image thecamera identifier, authorship identifier, image owner identifier, and/orimage identifier. The accounting application automatically determinesthe appropriate compensation accruing to the owner and/or author of theimage by accessing a customer database which stores such compensationinformation in association with the camera identifier, authorshipidentifier, image ownership information, and/or image identifier.

The images may have been posted to the Web site by the image owner or byother users who have permission to do so. In order to protect theowner's rights in the image, if the image has been posted without theowner's permission, the owner can request that the image be removed fromthe Web site.

A fixed price may be charged for each image downloaded or exchanged.Optionally, users may be charged different fees for different images. Inthis example, the customer database stores the image identifier and usefee for the image in association with the image owner's identifier.Thus, each time the image is downloaded and exchanged, the accountingapplication utilizes the image owner identifier and/or the unique imageidentifier as a key to retrieve the use fee associated with the imageand to then store in association with the image owner identifier theamount accrued as a result of the download, copying, or exchange of theimage. In addition, the user downloading or copying the image will becharged the corresponding use fee. The user may have a prepaid accountwhich can be debited, a credit card that can be charged, or may simplybe billed on a periodic basis for downloaded or copied images.

Further, users may optionally be charged different flat fees for imagestaken by 15 corresponding different photographers. Thus, for example,users may be charged more for images taken by selected well knownphotographers than for images taken by relatively unknown photographers,regardless of who the image owner is. Therefore, the accountingapplication retrieves the use fee for a given image by using theauthor's or photographer's identifier as a key.

Alternatively, users may be charged a flat fee to download, copy orexchange a predetermined number of images, with additional fees forimages over and above the predetermined number of images. Differentusers may have different use plans with different set limits and fees.For example, one user may be charged $10 a month for the right todownload 100 pictures a month and may be charged an additional fee forevery image downloaded over the 100 picture limit, such as 15 cents apicture. Another user may be charged $100 a month for the right todownload 2000 pictures a month and may be charged an additional fee forevery image downloaded over the 2000 picture limit, such as 7 cents apicture. The image owner may still be compensated for each downloaded orcopied image on an image-by-image basis, or may be paid a flat fee foruse of all or a part of the image owner's image library.

in this example, when the user accesses the Web site used to host orexchange the images, a user identifier associated with the user isobtained. The user identifier may be obtained by, for example,retrieving a user ID from a cookie stored on the user's Internetterminal or obtained via a log-in process. The accounting applicationutilizes the user identifier as a key to retrieve user accountinformation from the customer database to determine what type of useplan the user has and what, if any, additional fee the user is to becharged for an image. The accounting application utilizes the cameraidentifier and/or, owner identifier embedded in the image to locateaccount information for the image owner and to appropriately credit theimage owner's account for the download, copying or other use of theimage.

If an image contains both a camera identifier and an image owneridentifier, one embodiment of the present invention performs across-check to better ensure that someone has not improperly embedded animage owner identifier that does not correspond to the actual imageowner. The camera identifier is extracted from the image watermark. Theidentifier is used as a key to access the identity of the registeredcamera owner from the camera registry database previously described. Ifthe camera owner identifier is not the same as or does not correspond tothe image owner identifier extracted from the image, one or moresafeguard procedures are performed to better ensure the image is notimproperly posted, exchanged, or used and/or to further ensure anycredits associated with the downloading, exchange or other transfer ofthe image is applied to the correct customer account. Thus, the “theft”of an image by improperly embedding inaccurate ownership information isavoided.

For example, if the registered camera owner identity is not the same asthe image owner identity extracted from the image, the registered cameraowner may be contacted and asked whether the camera is now owned by theuser claiming ownership of the image, and if the camera ownership hastransferred, when the transfer occurred. If the registered camera ownerindicates that ownership of the camera has indeed been transferred tothe user claiming ownership of the image, the user claiming ownership ofthe image is contacted via regular mail or e-mail and asked if the userwould like to register as the new owner of the camera. Once the userregisters and establishes an account, credits for sales of the user'simages are posted to the user's account Optionally, it may be assumedthat images dated after the camera ownership transfer occurred belong tothe current owner of the camera, even though the image ownershipidentifier corresponds to the new camera owner and the camera owneridentifier corresponds to the old camera owner.

If, instead, the registered camera owner indicates that she or he stillowns the camera, the image will not be posted without the registeredcamera owner's permission and/or without proof that the user owns theimage. Additionally, the registered camera owner's identity is alsoembedded in the image as the image owner's identity, and revenues thataccrue as a result of transactions involving the image are posted to theregistered camera owner's account. Optionally, the registered cameraowner may be required to prove that he or she still owns or controls thecamera. For example, the registered camera owner may be required to takea photograph of a time sensitive item, such as a current newspaper, andprovide it to the Web site operator. The image is examined and thecamera identifier is extracted. If the extracted camera identifiermatches the camera identifier extracted from the original imagesubmitted by the user, then the registered camera owner has demonstratedownership of the original image. Alternatively or in addition, the userwho had claimed ownership in the original image may also be providedwith the opportunity to demonstrate ownership or control of the cameraused to capture the image or otherwise demonstrate ownership of theimage.

Thus, as described above, the present invention advantageously embeds inimages information about the characteristics and/or settings of theimage capture device used to capture the image. This information may beused to enhance reproductions of the captured images using a givenreproduction device, as well as to identify the source of the image.Further, in one embodiment, information may be embedded using awatermark encrypted using a key based at least in part on the cameracharacteristics. In addition, identifiers embedded in or associated withimages are used to meter the use or transfer of images.

Although this invention has been described in terms of certain preferredembodiments, other embodiments that are apparent to those of ordinaryskill in the art are also within the scope of this invention.

1. A camera, comprising: an imaging device configured to capture animage, wherein an inherent characteristic associated with the imagingdevice affects an image quality of every image that the imaging devicecaptures; and a processing device configured to apply a watermark to theimage, wherein the watermark indicates the inherent characteristicassociated with the imaging device, and wherein the watermark uniquelyidentifies the camera.
 2. The camera of claim 1, wherein the inherentcharacteristic comprises a distribution pattern associated with one ormore defective pixels of the imaging device.
 3. The camera of claim 2,wherein the distribution pattern indicates both a number of thedefective pixels and a location of the defective pixels in the imagingdevice.
 4. The camera of claim 1, wherein the watermark comprises anidentical watermark that is applied to all the images captured by theimaging device.
 5. The camera of claim 4, wherein the identicalwatermark indicates copyright ownership of all the images captured bythe imaging device.
 6. The camera of claim 1, wherein the inherentcharacteristic comprises at least one of: color cross-talk information,pixel gain values, pixel row offset, pixel column offset, or currentleakage.
 7. The camera of claim 1, wherein the processing device isfurther configured to encrypt the watermark utilizing the inherentcharacteristic.
 8. An apparatus, comprising: means for capturing animage, wherein an inherent characteristic of the means for capturingsimilarly affects an image quality of every image that is captured; andmeans for applying a watermark to the image, wherein the watermarkindicates the inherent characteristic, and wherein the watermarkuniquely identifies the apparatus.
 9. The apparatus of claim 8, whereinthe inherent characteristic comprises a distribution pattern associatedwith one or more pixels of the imaging device.
 10. The apparatus ofclaim 9, wherein the one or more pixels comprise a number of pixelsproviding a response outside of a desired range, and wherein thedistribution pattern indicates a location of the number of pixels in theimaging device.
 11. The apparatus of claim 9, wherein the means forapplying comprises means for applying the watermark to all the imagescaptured by the imaging device.
 12. The apparatus of claim 11, whereinthe watermark comprises an identical watermark that is applied to allthe images captured by the imaging device.
 13. The apparatus of claim12, further comprising means for associating the identical watermarkwith copyright ownership of all the images captured by the imagingdevice.
 14. A method, comprising: capturing an image with an imagecapturing device, wherein an inherent characteristic of the imagecapturing device affects an image quality of every image that iscaptured by the image capturing device; and applying, with the imagecapturing device, a watermark to the image, wherein the watermarkindicates the inherent characteristic, and wherein the watermarkuniquely identifies the image capturing device.
 15. The method of claim14, wherein the inherent characteristic comprises a distribution patternassociated with one or more defective pixels of the image capturingdevice.
 16. The method of claim 15, wherein the distribution patternindicates a location of the one or more defective pixels in the imagecapturing device.
 17. The method of claim 14, wherein the inherentcharacteristic comprises at least one of: sensor frequency response,sensor current value, or sensor sensitivity.
 18. The method of claim 14,further comprising applying the watermark to all the images captured bythe image capturing device.
 19. The method of claim 14, furthercomprising associating the watermark with one or more copyrights to theimage.
 20. The method of claim 19, further comprising: receiving theimage; analyzing the watermark; and identifying an ownership of the oneor more copyrights to the image in response to analyzing the watermark.